Skip to content

feat: support incremental MCI add maintenance#2087

Open
LightWant wants to merge 10 commits into
antgroup:mainfrom
LightWant:feat/filter-MCI
Open

feat: support incremental MCI add maintenance#2087
LightWant wants to merge 10 commits into
antgroup:mainfrom
LightWant:feat/filter-MCI

Conversation

@LightWant

@LightWant LightWant commented May 20, 2026

Copy link
Copy Markdown
Collaborator

Change Type

  • Bug fix
  • New feature / behavior extension
  • Improvement/Refactor
  • Documentation
  • CI/Build/Infra

Linked Issue

N/A (kind/improvement)

What Changed

  • Implemented incremental Add-time clique maintenance for MCI.
    • Build(prefix) can now be followed by Add(tail) without clearing the clique index.
    • New clique memberships are stored in a lazy delta overlay instead of rewriting the base CSR clique index.
  • Added configurable incremental clique-join controls:
    • join_ratio_threshold, default 0.6
    • added_mct, default 3
  • Updated MCI serialization/deserialization to persist and restore the embedded HGraph payload.
    • hgraph_index_path remains available as an external fallback for older or split-artifact workflows.
  • Added eval tools for incremental-vs-full MCI experiments and index comparison.
  • Added a runnable C++ example: examples/cpp/323_feature_mci_incremental_add.cpp.
  • Added English documentation: docs/docs/en/src/advanced/mci_incremental_add.md.

Test Evidence

  • make fmt
  • make lint
  • make test
  • make cov, run tests, and collect coverage
  • Other (describe below)

Test details:

git diff --check
./build/tests/unittests -d yes '[MCIParameter]' --allow-running-no-tests ''
./build/tests/unittests -d yes '[mci]' --allow-running-no-tests ''
./build/tests/functests -d yes '[mci]' --allow-running-no-tests ''
cmake -B build-examples -DENABLE_EXAMPLES=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-examples --target 323_feature_mci_incremental_add -j$(nproc)
./build-examples/examples/cpp/323_feature_mci_incremental_add

Example output:

Built prefix vectors: 128
Incrementally added vectors: 128
Serialized MCI index with embedded HGraph: /tmp/vsag_mci_incremental_example/mci_incremental.index
Reloaded hybrid route: "hgraph"

Experiment Notes

wufufilter-5m incremental reload:
  recall_avg=0.9899
  qps=17918.7

codefilter-3m incremental reload:
  recall_avg=0.9120
  qps=952.5

codefilter-3m existing full-build baseline comparison:
  full_c50 recall_avg=0.8577 qps=1007.7
  incr_c20_jr06_mct3 recall_avg=0.9120 qps=1032.1

Compatibility Impact

  • API/ABI compatibility: no public C++ API signature changes.
  • Parameter compatibility: old MCI configs remain valid; new parameters have defaults.
  • Serialization compatibility: old MCI indexes without lazy delta or embedded HGraph payload still deserialize; external hgraph_index_path fallback remains supported.

Performance and Concurrency Impact

  • Incremental Add avoids O(N) CSR middle inserts for clique membership updates by writing delta overlay rows.
  • Search reads both base clique CSR and delta overlay memberships.
  • No new concurrency guarantee beyond the existing MCI Add/search locking behavior.

Documentation Impact

  • No docs update needed
  • Updated docs:
    • README.md
    • DEVELOPMENT.md
    • CONTRIBUTING.md
    • Other: docs/docs/en/src/advanced/mci_incremental_add.md, docs/docs/en/src/indexes/mci.md, examples/cpp/README.md

Risk and Rollback

  • Risk level: medium
  • Rollback plan: revert commit 45102600.

Checklist

  • I have linked the relevant issue or marked it N/A for kind/improvement
  • I have added/updated tests for new behavior or bug fixes
  • I have considered API compatibility impact
  • I have updated docs if behavior/workflow changed
  • My commit messages follow project conventions (Conventional Commits, DCO trailer, Assisted-by trailer)

Copilot AI review requested due to automatic review settings May 20, 2026 07:16
@LightWant LightWant added kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 version/0.18 labels May 20, 2026
@mergify

mergify Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 Require kind label

  • label~=^kind/

🟢 Require version label

  • label~=^version/

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the MCI index, a dense-vector index utilizing maximal-clique candidate structures and an optional HGraph hybrid overlay for filtered searches. The implementation includes the core algorithm, parameter handling, comprehensive documentation, and benchmark configurations. Feedback identifies a duplicated key in a benchmark YAML file and suggests using YAML anchors to manage configuration duplication. Additionally, the reviewer recommends defining variables in documentation code snippets for clarity, replacing std::getenv and std::cerr with more robust configuration and logging mechanisms, allowing a limited_size of zero in range searches, and refactoring duplicated result-set creation logic into a helper function.

Comment thread benchs/datasets/wufufilter_5m_mci_self_build_search_sweep.yml Outdated
Comment thread benchs/datasets/codefilter_3m_hgraph_mci_hybrid_compare_search.yml Outdated
Comment thread docs/docs/en/src/indexes/mci.md
Comment thread docs/docs/en/src/indexes/mci.md Outdated
Comment thread docs/docs/zh/src/indexes/mci.md
Comment thread docs/docs/zh/src/indexes/mci.md Outdated
Comment thread src/algorithm/mci.cpp Outdated
Comment thread src/algorithm/mci.cpp Outdated
Comment thread src/algorithm/mci.cpp Outdated
Comment thread src/algorithm/mci.cpp Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the new MCI dense-vector index type (with optional hybrid routing to an external HGraph for broad filtered searches), plus evaluation tooling/docs/examples to support benchmarking and adoption across the repo.

Changes:

  • Introduces the MCI index implementation, parameter parsing, factory registration, and unit tests.
  • Updates the eval pipeline for filtered searches that may return fewer than topk, and adds a KNNG export tool.
  • Adds C++ example and English/Chinese documentation + benchmark YAML presets for MCI and MCI/HGraph hybrid workflows.

Reviewed changes

Copilot reviewed 47 out of 47 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tools/eval/monitor/recall_monitor.cpp Updates recall computation to handle filtered results returning fewer than topk.
tools/eval/export_knng.cpp Adds a CLI tool to export a fixed-width KNNG binary by running HGraph searches over base vectors.
tools/eval/CMakeLists.txt Builds and links the new export_knng executable and ensures OpenMP flags are applied.
tools/eval/case/search_eval_case.h Adds cached valid-id lists to support Filter::GetValidIds() in eval runs.
tools/eval/case/search_eval_case.cpp Enables parallel filtered KNN eval and supplies result count + valid-id hints to monitors/filters.
src/inner_string_params.h Adds internal string constant for mci index type.
src/factory/index_creators.cpp Wires MCI into the index factory registration.
src/factory/factory_test.cpp Adds factory coverage to create an MCI index from full parameters.
src/constants.cpp Exposes INDEX_MCI constant for API/config usage.
src/algorithm/mci.h Declares the MCI index class and its hybrid-search hooks.
src/algorithm/mci.cpp Implements MCI build/search/serialize/deserialize, clique enumeration, and hybrid routing to HGraph.
src/algorithm/mci_test.cpp Adds unit tests for MCI build, filtered search, RabitQ one-bit mode, KNNG import, and hybrid overlay.
src/algorithm/mci_parameter.h Defines MCI build/search parameter schemas (including hybrid overlay settings).
src/algorithm/mci_parameter.cpp Implements MCI parameter parsing, validation, JSON conversion, and search param parsing.
src/algorithm/mci_parameter_test.cpp Adds parameter round-trip and compatibility tests for MCI (including hybrid settings).
include/vsag/index.h Extends public IndexType enum with MCI.
include/vsag/constants.h Declares INDEX_MCI in the public constants header.
examples/cpp/CMakeLists.txt Builds the new runnable example for MCI hybrid filtered search.
examples/cpp/322_feature_mci_hybrid_filter.cpp Demonstrates exporting KNNG from HGraph, building MCI, and running hybrid filtered queries routed to HGraph.
docs/docs/zh/src/SUMMARY.md Adds MCI page to Chinese docs navigation.
docs/docs/zh/src/resources/index_parameters.md Documents MCI build/search parameters in Chinese.
docs/docs/zh/src/indexes/README.md Adds MCI to Chinese index overview table.
docs/docs/zh/src/indexes/mci.md Adds Chinese MCI documentation page (incl. hybrid overlay and KNNG format).
docs/docs/zh/src/guide/create_index.md Lists mci in Chinese create-index guide.
docs/docs/en/src/SUMMARY.md Adds MCI page to English docs navigation.
docs/docs/en/src/resources/index_parameters.md Documents MCI build/search parameters in English.
docs/docs/en/src/indexes/README.md Adds MCI to English index overview table.
docs/docs/en/src/indexes/mci.md Adds English MCI documentation page (incl. hybrid overlay and KNNG format).
docs/docs/en/src/guide/create_index.md Lists mci in English create-index guide.
benchs/datasets/wufufilter_5m_mci_self_build.yml Adds benchmark preset for self-built MCI on WUFUFILTER 5M.
benchs/datasets/wufufilter_5m_mci_self_build_sq8.yml Adds benchmark preset variant using SQ8 base quantization.
benchs/datasets/wufufilter_5m_mci_self_build_search_sweep.yml Adds WUFUFILTER 5M MCI search sweep presets.
benchs/datasets/wufufilter_5m_mci_hgraph_knng_search_sweep.yml Adds sweep presets for MCI built from an HGraph-derived KNNG.
benchs/datasets/wufufilter_5m_mci_hgraph_knng_build.yml Adds build+search preset for MCI using an HGraph-derived KNNG.
benchs/datasets/wufufilter_5m_mci_hgraph_hybrid_search_sweep.yml Adds sweep presets for hybrid MCI/HGraph routing experiments.
benchs/datasets/wufufilter_5m_hgraph_mci_hybrid_compare_search.yml Adds side-by-side comparison presets (HGraph vs MCI vs hybrid thresholds).
benchs/datasets/wufufilter_5m_hgraph_build.yml Adds baseline HGraph build preset for WUFUFILTER 5M.
benchs/datasets/wufufilter_5m_filtered_hgraph_search_sweep.yml Adds filtered-search sweep presets for HGraph on WUFUFILTER 5M.
benchs/datasets/gist1m_sq8_uniform_pure_baseline.yml Adds baseline gist1m SQ8-uniform HGraph benchmark presets.
benchs/datasets/gist1m_sq8_pure_baseline.yml Adds baseline gist1m SQ8 HGraph benchmark presets.
benchs/datasets/gist1m_base_quantization_sq8_vs_rabitq_build_search.yml Adds gist1m build+search comparison presets (SQ8 vs RabitQ).
benchs/datasets/codefilter_3m_mci_self_build.yml Adds codefilter 3M MCI benchmark preset(s).
benchs/datasets/codefilter_3m_mci_self_build_search_sweep.yml Adds codefilter 3M MCI search sweep presets.
benchs/datasets/codefilter_3m_ivf_build.yml Adds codefilter 3M IVF build preset for comparison.
benchs/datasets/codefilter_3m_hgraph_mci_hybrid_compare_search.yml Adds codefilter 3M comparison presets across HGraph/MCI/hybrid.
benchs/datasets/codefilter_3m_hgraph_build.yml Adds baseline HGraph build preset for codefilter 3M.
benchs/datasets/codefilter_3m_filtered_search_sweep.yml Adds filtered-search sweep reference presets for codefilter 3M.

Comment thread tools/eval/monitor/recall_monitor.cpp
Comment thread src/algorithm/mci.cpp
Comment thread src/algorithm/mci_parameter.cpp
Comment thread benchs/datasets/wufufilter_5m_mci_self_build_search_sweep.yml Outdated
Copilot AI review requested due to automatic review settings May 30, 2026 06:41
@LightWant LightWant force-pushed the feat/filter-MCI branch 2 times, most recently from 71bfab0 to 73f1641 Compare May 30, 2026 07:53

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid, well-structured addition of the MCI index. The clique enumeration, CSR layout, and HGraph-hybrid routing are coherent, and the load_clique_index path validates its CSR invariants carefully. A few correctness/safety concerns worth addressing before merge:

  1. Deserialize reads the four CSR arrays (p_maxc_, maxcs_, p_node_to_cid_, node_to_cids_) straight from the stream with no validation, unlike load_clique_index. search_clique_candidates then uses node_to_cids_[] to index visited_cliques/p_maxc_ and maxcs_[] to index node arrays. A corrupted or mismatched index file can therefore cause out-of-bounds reads. has_clique_index only checks array sizes, not the values inside them.
  2. The hybrid path is exclusive routing (whole query forwarded to HGraph), not score fusion — consistent with the docs, so no merging-correctness issue, but be aware there is no fallback if HGraph returns fewer than k filtered results.
  3. There is a data race between incremental Add (mutates p_node_to_cid_ under add_mutex_ only) and concurrent search/clear_clique_index (which touch the same vector under global_mutex_).

The rest is mostly minor. Tests are thorough (build, filter, reorder, concurrent, serialize, hybrid). (All flagged from the diff — please confirm against full file context.)

Comment thread src/algorithm/mci.cpp
Comment thread src/algorithm/mci.cpp Outdated
Comment thread src/algorithm/mci.cpp Outdated
Comment thread src/algorithm/mci.cpp
Comment thread src/algorithm/mci.cpp
Comment thread src/algorithm/mci.cpp
Comment thread src/algorithm/mci_parameter.cpp
@LightWant LightWant changed the title feat: add MCI hybrid workflow support feat: support incremental MCI add maintenance Jun 12, 2026
- add the MCI index, HGraph hybrid overlay, and external KNNG import path
- add eval/export tooling plus benchmark configs for filtered MCI comparisons
- document MCI in English and Chinese and add a runnable hybrid example

Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: GitHub Copilot:GPT-5.4
PR review fixes:
- docs (en/zh) mci.md: add missing variable definitions (n, ids, data, q)
  to the example snippets so they are self-contained and compilable.
- src/algorithm/mci.cpp: replace std::getenv("VSAG_MCI_BUILD_STATS") and
  std::cerr instrumentation with logger::debug so the standard logger
  level controls verbosity; relax the limited_size != 0 check; extract
  the duplicated heap-to-dataset construction into
  MCI::build_dataset_from_heap helper shared by KNN and range search.
- src/algorithm/mci.{h,cpp,_parameter.cpp}: drop the redundant
  hgraph_serialized_size field from Serialize/Deserialize; hybrid loading
  now relies solely on hgraph_index_path. MCIParameter::ToJson now
  serializes hgraph_index_path so JSON round-trips are complete.
- tools/eval/monitor/recall_monitor.cpp: align the result-tuple cast with
  SearchEvalCase's actual std::tuple<const int64_t*, ...> type, removing
  an undefined-behavior reinterpret_cast.

Tests:
- tests/test_mci.cpp: new functional test suite covering build/search
  across metrics and base quantization types, filter search, reorder
  (rabitq one-bit), concurrent KNN, the three serialization paths
  (binary set / reader set / file), the hybrid overlay path with a
  decoupled hgraph index loaded via hgraph_index_path, and a
  RandomAllocator robustness case.

Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: CodeFuse:claude-sonnet-4.5
clang-format-15 noted layout drift in four files introduced by the MCI
feature; this commit applies the formatter output verbatim so the
"check_format" CI job stops failing.

- examples/cpp/322_feature_mci_hybrid_filter.cpp
- src/algorithm/mci.cpp
- src/algorithm/mci_test.cpp  (also: append trailing newline)
- tools/eval/export_knng.cpp

Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: CodeFuse:claude-sonnet-4.5
…ecks

The CHECK_ARGUMENT macro expands to `if (not(expr)) { ... }`; when
`expr` itself is a compound `A and B` (or `not X or Y`) clang-tidy's
readability-simplify-boolean-expr treats the resulting AST as
DeMorgan-reducible and fails the CI lint job at -warnings-as-errors.

Split the two affected MCI checks onto two lines and trail the first
clause with `// NOLINT`, mirroring the existing pattern in
src/algorithm/hgraph/hgraph_param_mapping.cpp:488.

Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: CodeFuse:claude-sonnet-4.5
Resolve the 7 clang-tidy-15 errors reported by the CI lint job
(`readability-*` and `bugprone-narrowing-conversions`):

- L54  readability-static-definition-in-anonymous-namespace:
       drop redundant `static` from MCI_PARAMS_TEMPLATE (already in
       anonymous namespace).
- L217 readability-suspicious-call-argument: silence with NOLINT on
       the symmetric `set_core_edge(rhs, lhs)` companion call; this is
       intentional for undirected edge insertion.
- L453 readability-identifier-naming: rename struct
       `MCISearchCandidate` -> `mci_search_candidate` to match the
       project's `StructCase: lower_case` rule. Updated all 4 use sites.
- L627 bugprone-narrowing-conversions: cast `extra_info_size_`
       (`uint64_t`) to `int64_t` when calling `make_temporary_sq8_flatten`,
       mirroring `hgraph_build.cpp:147`.
- L708, L1939 readability-simplify-boolean-expr: split each
       compound `CHECK_ARGUMENT` predicate across two lines and trail
       the first clause with `// NOLINT`, same pattern as
       `hgraph_param_mapping.cpp:488` and `mci_parameter.cpp` (this PR).
- L1891 readability-qualified-auto: declare `vectors` as
       `const auto*` (the function already returns `const float*`).

Verified locally with clang-tidy-15 (0 errors/warnings on mci.cpp +
mci_parameter.cpp) and `build/tests/functests "[mci]"` passes 21173
assertions across 7 test cases.

Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: CodeFuse:claude-sonnet-4.5
Add lazy clique maintenance for incremental MCI Add, persist embedded HGraph state, and add evaluation utilities for full-build versus incremental comparisons.

Document the incremental flow and add a runnable C++ example.

Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: Codex:GPT-5
Add validate_clique_csr() to check consistency of deserialized CSR
arrays (monotonicity, tail==size, ID ranges), matching the validation
already done in load_clique_index. Add defense-in-depth bounds checking
in collect_node_clique_ids to skip out-of-range clique IDs. Also fix
clang-format issues in mci_incremental_exp.cpp.

Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: ClaudeCode:claude-opus-4-6
- Fix data race in add_dataset: acquire global_mutex_ when mutating
  p_node_to_cid_ to prevent concurrent read/write with search threads.
- Add empty-heap early return in build_dataset_from_heap to avoid
  unsigned underflow when range search yields no results.
- Add debug log when hybrid routing is silently disabled due to
  hgraph/mci size mismatch after incremental Add.
- Document best-effort coverage semantics of try_select_clique under
  parallel builds.

Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: ClaudeCode:claude-opus-4-6
Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: ClaudeCode:claude-opus-4-6
Avoid clang-tidy boolean simplification diagnostics in MCI CSR validation and remove the duplicate export_knng eval target.

Signed-off-by: zhuangye.yxw <2510035537@qq.com>
Assisted-by: Codex:GPT-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 module/api module/docs module/example module/testing module/tools size/XXL version/0.18

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants